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Abstract 


Data  Envelopment  Analysis  (DEA)  is  a  linear  programming  based  technique 
that  was  developed  to  evaluate  relative  efficiency  of  nonprofit  and  public 
sector  Decision  Making  Units  (DMU's)  that  use  multiple  inputs  to  produce 
multiple  outputs.  In  this  study,  DEA  is  evaluated  and  tested  for  use  as  a 
managerial  audit  tool  to  identify  and  measure  inefficiencies  among  a  set  of 
DI'IU's.   Based  on  three  applications  of  DEA,  this  technique  is  found  to  be  a 
useful  technique  for  allocation  of  audit  resources  and  for  analytic  review  of 
operating  efficiency  when  applied  to  a  specific  set  of  audit  situations  and 
when  interpreted  with  recognition  of  DEA's  particular  strengths  and 
limitations.  The  value  of  DEA  is  further  found  to  extend  to  a  class  of  for 
profit  managerial  audits  in  addition  to  the  nonprofit  and  public  sector  types 
of  audits. 


Managerial  audits  designed  to  evaluate  the  effectiveness  and  efficiency  of 
the  operations  of  an  organization  have  gained  increased  acceptance  and  have 
been  increasingly  used  by  managers  in  government  and  business.  These 
managerial  audits,  also  referred  to  as  "comprehensive  audits"  [7],  operational 
audits",  "operation  reviews"  [9],  were  first  actively  used  by  the  U.S.  General 
Accounting  Office  [11]  and  have  since  been  utilized  by  other  governments, 
e.g.,  Canada,  Australia,  and  Israel  as  well  as  by  various  state  audit 
agencies.  Corporate  internal  auditors  have  increasingly  been  required  to 
complete  managerial  audits  [9]  and  regulatory  agencies  have  hired  management 
consulting  firms  to  conduct  such  audits  for  utilities  (see  for  example  [12]). 
It  has  been  suggested  by  J.  Burton  [1]  that  managerial  audits  should  be 
included  as  an  integral  part  of  a  financial  audit  by  CPA's  to  increase  the 
value  of  their  service  and  to  better  justify  the  increasing  cost  of  such 
audits  to  management.  Trends  toward  increased  use  of  managerial  audits 
suggest  that  any  methodologies  which  help  to  achieve  the  objectives  of  these 
audits  will  be  of  value. 

In  this  paper  the  use  of  Data  Envelopment  Analysis  is  evaluated  as  a 
managerial  audit  technique.  Data  Envelopment  Analysis  (DEA)  is  a  linear 
programming  based  technique  developed  by  A.  Charnes,  W.  W.  Cooper,  and 
E.  Rhodes  (CCR)  [5]  [6]  to  evaluate  the  relative  efficiency  of  public  sector 
Decision  Making  Units  (DMU's)  that  use  multiple  inputs  to  produce  multiple 
outputs.  The  mathematical  integrity  of  DEA  and  its  consistency  with 
microeconomic  theory  have  been  documented  by  CCR  [5]  [6].  Relying  on  the 
soundness  of  this  theoretical  foundation,  DEA  has  been  used  to  evaluate 
various  public  sector  DMU's  such  as  educational  institution  [2]  [3]  and  armed 
forces  recruiting  office  [8].  The  purposes  is  to  investigate  how  DEA  results 
can  be  interpreted  and  used  in  a  managerial  audit  context  to  evaluate  the 
efficiency  of  DMU's  and  to  define  the  application  where  DEA  is  most 


appropriate  compared  with  more  traditional  audit  technique  to  assess 
organization  efficiency.  In  addition,  this  investigation  serves  to  clarify 
and  illustrate  the  strengths  and  weaknesses  of  DEA  and  suggests  that  DEA  can 
be  effectively  applied  in  many  for  profit  business  settings  for  managerial 
audit  purposes  in  addition  to  its  original  intended  use  for  public  sector  and 
other  nbnprofit  organization  evaluations. 

Efficiency  versus  Effectiveness  -  Defined 

Before  proceeding,  I  should  clarify  the  types  of  audit  objectives  for 
which  we  consider  use  of  DEA  in  this  paper.  The  managerial  audit  may  attempt 
to  evaluate  effectiveness,  the  ability  of  a  DMU  to  set  and  meet  goals,  and 
efficiency,  the  use  of  inputs  to  produce  the  desired  outputs.  I  do  not 
consider  the  effectiveness  objective  in  this  paper  but  rather  assume  that  the 
ouptputs  selected  by  the  DMU  are  consistent  with  their  effectiveness  criteria, 
i.e.,  that  they  are  producing  goods  or  services  that  are  consistent  with  the 
goals.  Rather,  the  emphasise  is  on  the  assessment  of  DMU's  technical 
efficiency.  A  DMU  is  defined  here  to  be  technically  inefficient  if  a)  the  DMU 
could  produce  the  same  level  of  the  outputs  it  produced  with  fewer  inputs  than 
it  used  or  b)  the  DMU  could  have  produced  more  outputs  than  it  produced  with 
the  same  level  of  inputs  used. 

Efficiency  Evaluation  of  multiple  output-multiple  input  organizations 
The  characteristics  of  DEA  that  prompt  interest  in  evaluating  public 
sector  and  nonprofit  organizations  are  as  follows 

1.  Ability  to  simultaneously  consider  multiple  outputs  and  inputs  in 
evaluating  efficiency. 

2.  The  production  function,  i.e.,  efficient  input-output  relationship 
need  not  be  known. 

These  characteristics  are  particularly  useful  for  nonprofit/public  sector 
evaluation  because  such  organization  produce  multiple  outputs  which  cannot  be 


adequately  evaluated  with  more  traditional  measures  such  as  profit  and  return 
or  investment,  as  profit  maximization  or  cost  minimization  is  but  one  of  a 
broader  set  of  goals.  For  example,  a  nonprofit  hospital  may  produce  multiple 
ouptputs  including  treatment  of  a  variety  of  patient  types,  research,  training 
nurses  and  medical  students,  and  community  education.  Most  of  these  outputs 
do  not  have  competitive  market  prices  and  the  amount  of  inputs  needed  to 
efficiently  produce  these  outputs  is  generally  not  known  in  any  detail.  An 
evaluation  of  hospital  efficiency  needs  to  consider  the  amount  of  resources 
used  to  provide  all  these  outputs  when  the  efficient  output  input 
relationships  are  not  known.  Hence,  DEA  appears  to  be  well  suited  to  evaluate 
the  efficiency  of  such  organizations  and  consequently  it  is  evaluated  as  a 
managerial  audit  tool  for  such  applications.  The  applicability  of  DEA  in  a 
for  profit  audit  application  is  also  considered  at  a  latter  point  in  the  paper. 

Outline  of  the  paper 

The  following  section  2  briefly  describes  the  DEA  technique  and  ways  it 
can  be  applied  using  standard  linear  programming  computer  codes.  Section  3 
describes  an  application  of  DEA  to  an  artificial  data  set  where  the  efficient 
and  inefficient  DMU's  are  known  a  priori.  We  use  this  artificial  data  set  to 
investigate  DEA's  ability  to  identify  inefficient  DMU's  compared  to  the  known 
inefficient  DMU's.  This  approach  is  adopted  because  in  field  application  of 
DEA  as  in  [3]  and  [8]  the  truly  inefficient  DMU's  are  not  known  and  hence,  the 
accuracy  of  the  DEA  results  cannot  strictly  be  evaluated  as  I  attempt  to  do 
here.  When  DEA  locates  inefficient  DMU's,  these  DMU's  are  strictly 
inefficient  compared  to  other  DMU's  in  the  observation  set.  However,  I  also 
find  that  even  in  this  somewhat  simple  example,  all  the  inefficient  DMU's  are 
not  identified  as  such.  In  addition,  comparing  the  detailed  DEA  results  to 
the  known  inefficiencies  indicates  alternative  paths  to  improving  efficiency 


-3- 


of  inefficient  DMU's  but  it  does  not  single  out  the  specific  path  that 
directly  moves  the  DMU  to  the  "known"  efficient  production  function. 

Section  4  describes  the  field  testing  of  DEA  in  a  set  of  teaching 
hospitals  where  the  inefficient  hospitals  are  not  known,  as  will  be  the  case 
in  most  applications  of  efficiency  evaluation  techniques  like  DEA.  A  panel  of 
hospital  experts,  including  the  management  of  one  of  the  identified 
inefficient  hospitals,  are  used  to  evaluate  the  DEA  results.   In  addition,  the 
DEA  technique  is  compared  with  the  widely  used  ratio  analysis  approach  to 
determine  its  relative  capabilities  in  identifying  inefficient  DMU's.  The  DEA 
results  are  found  to  be  accurate  based  on  the  experts  Judgment.   In  addition, 
the  traditional  use  of  ratio  analysis  is  found  to  be  incapable  of  locating 
inefficiencies  identified  via  DEA. 

Section  5  discusses  an  application  of  DEA  in  a  for  profit  setting  and 
considers  ways  that  this  technique  may  be  appropriate  in  this  sector.  The 
branches  of  a  savings  bank  are  evaluated  using  DEA  and  the  results  suggest 
ways  in  which  DEA  may  be  useful  in  a  managerial  audit  of  businesses  with 
multioffice  organization  and  which  need  to  consider  multiple  outputs  and 
inputs  in  their  evaluation  of  operating  efficiency. 

Section  6  concludes  by  discussing  how  DEA  might  be  appropriately  used  as  a 
managerial  audit  tool,  how  it  relates  to  other  traditional  techniques  with 
respect  to  their  relative  strengths  and  weaknesses  and  the  areas  that  need  to 
be  researched  in  further  developing  this  tool  for  managerial  audit  application. 

2.  Data  Envelopment  Analysis  -  a  new  methodology  for  measurement  of 
relative  technical  efficiency. 

Data  Envelopment  Analysis  generalizes  the  usual  output  to  input  ratio 

measure  of  technical  efficiency  in  terms  of  a  fractional  linear  program 

which  can  be  summarized  as  follows  [5]  [6]. 
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Objective: 
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where  o  is  the  Decision  Making  Unit  (DMU)  being 
evaluated  in  the  set  ofj  =  l n  DMU's. 


Constaints ; 


Less  than 

Unity 

Constraints 


Positivity 
Constraints 

Data: 


Outputs: 
Inputs: 


1  > 


Z  u  y  . 
1  r-'rj 


Z  v.x.  . 
i=l  ^  ^J 

0  <  Uj.  ;  r  =  1 ,  . 
0  <  Vi  ;  i  =  1,  . 


th 


.th 


y  .  =  observed  amount  of  r   output  for  the  j   DMU 
X.  .  =  observed  amount  of  i   input  for  the  i   DMU 


(1) 


The  data  used  for  each  DMU  are  the  y  .  outputs;  and  the  inputs,  x,  .. 

The  u  ,  V.  values  are  objectively  determined  from  the  data  in  terms  of 

the  above  model.  DEA  is  designed  for  an  ex  post  evaluation  of  how  efficient 

each  DMU  was  with  the  actual  inputs  (x.)  used  to  produce  its  outputs 

(y.)  without  explicit  knowledge  of  the  input-output  relationships  it 

used.  The  weights  in  the  form  of  the  u  and  the  v.  are  also  not  known 
&  r         1 

or  given  a  priori.  They  are,  instead,  calculated  as  (u  ,  v.)  values  to 

be  assigned  to  each  input  and  output  in  order  to  maximize  the  efficiency 

* 

rating  — h  —  of  the  DMU  being  evaluated.   That  is,  the  solution  sought 

is  the  set  of  (u  ,  v  )  values  that  will  give  the  DMU  being  rated  the 

* 

highest  efficiency  ratio,  h  ,  but  not  result  in  an  input-output  ratio 

exceeding  1  when  applied  to  any  DMU  in  the  observation  set. 
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Applying  DEA  to  a  set  of  DMU's  results  in  an  efficiency  rating  for  each 
DMU  of  1  (relatively  efficient)  or  less  than  1  (relatively  inefficient). 
These  ratings,  however,  represent  relative  efficiencies  obtained  from  a 
piecewise  linear  production  possibility  frontier  comprised  of  the  most 
efficient  unites  in  the  set  of  j  =  1 ,  .  .  . ,  n  DMU's. 

The  above  formulation,  involves  a  nonlinear-nonconvex  programming 
problem.  As  is  shown  in  [5]  however,  it  may  be  restated  into  a  dual  linear 
programming  problem  (which  can  be  used  with  any  linear  programming  computer 
code)  as  follows: 


s 

•  = 


Subject  to 


Max  h'  =  Z  0)  y 

o   ^^^  r^ro 

s  m 

0  >  I  y  V  .  -  Z  w.x     ;   j  =  1, 
r=l  ^  ^^         i=l  ^  ^J 


m 


Z  w.x.  (2) 


i=l   ^  ^° 


o  <  vij.,  uji  ;  r  -  1,  .  .  . ,  s 
1=1,  .  .  .,m 


In  order  to  include  the  condition  that  y  and  o),  are  strictly 
greater  than  zero,  it  requires  that 

G  <  Pi-'  e  <  coj  all  r  and  1  (3) 

where  0  <  e  is  a  positive  constant  which  is  so  small  that  it  cannot 
otherwise  disturb  any  solution  involving  only  real  numbers.   (Choice  of  an  e 
value  is  further  discussed  in  [6]  [14]). 

Note  that  to  apply  DEA,  we  only  need  to  identify  and  obtain  the  relevant 
set  of  output  and  input  data  and  that  each  output  and  input  need  only  be 
measured  in  its  natural  physical  units  without  the  need  to  use  a  homogeneous 
measurement  unit  like  dollars. 
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3,  Application  of  PEA  to  a  set  of  artificial  DMU's 
A  set  of  artiffcial  DMU's  where  a  simple  set  of  efficient  production 
relationships  was  hypothesized  to  examine  the  capabilities  of  DEA.  A  set  of 
fifteen  DMU's  was  created,  each  of  which  uses  three  Inputs  to  produce  various 
levels  of  3  outputs.   Seven  of  these  DMU's  used  the  exact  amount  of  the  three 
Inputs  to  jointly  produce  these  three  outputs  as  was  required  by  the 
hypothesized  efficient  production  function.  Eight  of  these  DMU's  used  more 
Inputs  than  was  required  by  the  efficient  production  function,  i.e.,  these 
eight  DMU's  were  known  to  be  inefficient.  For  convenience,  the  DMU's  were 
given  output  and  input  names  as  would  be  found  In  a  hospital.  The  artificial 
hospital  data  consisted  of  the  following  three  outputs  produced  with  the 
following  three  inputs  during  a  one  year  period  of  time. 

Outputs  Inputs 

y^    Regular  patients  treated  per  year    x^  Full  time  equivalent  staff 
(Reg.  Pat.'s)  used  per  year  (FTE's) 

y2    Severe  patients  treated  per  year     X2  Number  of  Hospital  bed  day 

(Sev.  Pat.'s)  available  per  year  (Bed  days) 

y3    Teaching  of  residents  and  interns    X3  Supplies  in  terms  of 
in  one  year  (Teach  units)  dollars  cost  per  year 

(Supply  i's) 

The  hypothesized  efficient  amount  of  Inputs  (x, ,  X2,  x,)  required 

for  each  output  (y-i,  yo*  Yt)  ^re  reported  in  Exhibit  1  in  the  appendix. 

Exhibit  2  in  the  appendix  reflects  the  data  base  that  was  developed  and  the 

inefficiencies  that  were  introduced  in  hospitals  H8  through  H15.  Exhibit  3  is 

an  example  of  how  the  data  in  Exhibit  2  was  developed  and  how  it  reflects  the 

efficient  production  relationships  hypothesized  in  Exhibit  1.  Exhibit  3,  for 

example,  compares  the  development  of  data  for  efficient  hospital  HI  and 

inefficient  hospital  H15.  We  began  by  arbitrarily  selecting  output  levels 
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(yi»  y2»  y^^   ^°^  each  hospital  and  arbitrarily  adding  inefficiencies  in 
hospitals  H8  through  H15.  H15  has  the  same  output  levels  as  HI  (see  Exhibits 
2  and  3),  but  H15  uses  an  inefficient  level  of  FTE's  (x,),  bed  days  (X2), 
and  supplies  (xt)  in  providing  treatment  for  regular  patients  (y,).  The 
inefficiencies  introduced  are  highlighted  by  the  circled  values  in  Exhibit  2. 
(The  development  of  the  data  base  is  further  discussed  in  [13]  and  [14]). 

The  efficiency  rating  results  from  DEA  are  summarized  in  table  1.  Note 
that  all  the  efficient  DMU's  (hospitals  HI  -  H7)  were  accurately  located  as 
relatively  efficient  (efficiency  rating  =  1.0).  Among  the  inefficient  DMU's 
(hospitals  H8  -  H15),  six  of  these  were  accurately  identified  as  inefficient 
and  two  were  not  located  as  inefficient  (HIO  and  H13),  This  result 
illustrates  the  primary  strength  and  limitation  of  DEA.  All  inefficient  DMU's 
located  will  be  strictly  inefficient,  i.e.,  there  are  other  DMU's  In  the 
observation  set  which  are  more  efficient  than  those  identified  as  inefficient 
via  DEA.  At  the  same  time,  DEA  will  not  necessarily  locate  all  the 
inefficient  units,  because  it  is  locating  inefficiency  by  comparing  DMU's 
within  the  entire  data  set.   Indeed,  there  may  be  no  technically  efficient 
units  in  the  data  set.   In  the  example  in  table  1  we  know  that  HI  -  H7  are 
technically  efficient  but  we  would  not  generally  know  this  when  using  a 
technique  like  DEA,  as  the  class  of  DI-IU's  where  DEA  may  be  most  helpful  are 
those  where  the  production  function  is  not  well  defined.  If  the  production 
function  were  known,  other  techniques  for  evaluating  efficiency  would  be  more 
direct  and  more  appropriate. 
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Table  1 

DEA  Efficiency 

Efficiency 

Efficient 

DMU's 

Rating  (E) 

Reference  Set 

HI 

1.0 

H2 

1.0 

H3 

1.0 

H4 

1.0 

H5 

1.0 

H6 

1.0 

H7 

1.0 

DEA 

Efficiency 

Efficiency 

Inefficient 

DMU's 

Rating  (E) 

Refei 
H4 

rence  Set 

H8 

0.99 

H9 

0.98 

HI, 

H2,  H6 

HIO 

1.0 

Hll 

0.85 

H4, 

H7 

H12 

0.99 

HI, 

H4,  H6 

H13 

1.0 

H14 

0.99 

HI, 

H4,  H6 

HI  5 

0.87 

H4. 

H6,  H7 
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The  value  of  DEA  with  respect  to  managerial  audits  is  that  the  units 
located  as  being  inefficient  are  clearly  inefficient  compared  to  other  DIlU's 
in  the  data  set  and  hence,  a  transfer  of  techniques  from  the  more  efficient 
units  to  the  less  efficient  units  can  potentially  improve  the  efficiency  of 
the  inefficient  Dt-IU's.  In  addition,  DEA  provides  information  which  helps 
locate  the  inefficiencies.   For  example,  Hll  is  found  to  be  inefficient 
(efficiency  rating  is  .85  which  is  less  than  1.0)  and  DEA  also  indicates  that 
the  inefficiency  is  located  most  directly  by  comparison  with  the  DMU's 
efficiency  reference  set,  H4  and  H7  as  indicated  in  table  1.  Hence,  the 
auditor  knows  that  Hll  is  Inefficient  and  that  its  inefficiency  can  be  studied 
most  directly  by  comparing  the  Hll  operation  wth  those  of  the  relatively 
efficient  DMU's,  H4  and  H7.   In  this  way,  DEA  focuses  the  search  for  sources 
and  remedies  for  inefficiency. 

One  issue  is  whether  other  analytic  techniques  are  already  adequate  and 
which  may  render  DEA  superfluous.  Using  this  same  artificial  data  set, 
Sherman  [14]  and  [13]  illustrates  that  the  use  of  ratio  analysis  and 
regression  econometric  techniques  were  less  powerful,  and  in  some  cases, 
misleading  in  identifying  inefficiencies  and  efficient  production 
relationships.  In  addition,  other  approaches  do  not  provide  insights  into  the 
magnitude  of  the  inefficiency  as  is  available  with  DEA. 

Beyond  the  location  of  inefficient  units,  DEA  provides  the  type  of 
information  illustrated  in  table  2  for  the  inefficient  DMU,  Hll.  This 
indicates  how  Hll  is  inefficient  compared  with  H4  and  H7  and  the  reduction  of 
inputs  that  would  make  Hll  efficient.   DEA  Indicates  that  a  weighted 
combination  of  H4  and  Hll  would  result  in  a  DMU  that  yields  as  much  or  more 
outputs  as  Hll  but  which  uses  less  inputs  (see  Column  E  in  table  2).  These 
weights  are  shadow  prices  directly  available  from  the  DEA  -  Linear  program 
results  and  are  further  described  in  [14]  and  [13].   Hence  we  learn  from  DEA 
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that  Hll  is  inefficient  and  the  amount  of  potential  input  reduction  may  be 
possible  without  decreasing  its  output  levels.   The  auditor  may  choose  to 
study  Hll  further  to  determine  the  reason  for  these  inefficiencies,  ways  these 
inefficiencies  can  be  reduced,  or  whether  there  are  policy  reasons  for 
maintaining  slack  resources. 

Based  on  our  knowledge  of  the  efficient  output-input  relationships,  the 
information  provided  in  table  2  must  also  be  qualified  with  respect  to  the 
degree  to  which  it  can  be  literally  interpreted.  DEA  results  in  table  2 
directly  indicate  that  a  combination  of  two  DMU's  operating  results  would 
produce  a  composite  DMU  that  is  more  efficient  than  Hll.  This  indicates  one 
way  for  Hll  to  become  more  efficient.  This  is  not,  however,  the  only 
direction  that  Hll  can  choose  or  should  choose  to  improve  efficiency.  For 
example,  Hll  was  inefficient  with  respect  to  its  use  of  FTE's  and  supplies 
used  to  provide  severe  and  regular  patients  care  (see  Exhibit  2).  The 
adjustment  required  to  make  Hll  efficient  based  on  the  known  production 
function  differs  from  the  adjustment  suggested  in  table  2  as  is  indicated  in 
table  3  below: 

Table  3 


A 

Actual  Inputs 

and 
outputs  of  Hll 

B 

Efficient  Inputs  for 
Hll  based  on  known 
production  function 

C 

Adjustment 
Required  to 
become  efficient 
(B  -  A) 

D 

Adjustment 

indicated  from 

table  2  based 

on  DEA  results 

^1 

44.5 

36.5 

-8 

-5.1 

^2 

65,260 

65,260 

0 

0 

^3 

265,000 

200,000 

-$65,000 

-45,710 

yi 

50 

+96.4 

^2 

3000 

0 

^3 

5000 

0 

-12- 


Table  3  compares  the  true  adjustment  required  by  the  efficient  production 
function  for  Hll  to  become  efficient  (Column  C)  with  the  DEA  results  (Column 
D) .  Column  C  in  table  3  reflects  the  adjustments  required  for  Hll  to  exactly 
fit  the  efficient  output-input  relationships  we  hypothesized.  The  two  sets  of 
adjustments  in  Columns  C  and  D  are  both  mathematically  accurate  ways  for  Hll 
to  become  efficient.  Note  however,  that  only  the  solution  in  Column  D  is 
available  directly  from  DEA  and  that  the  "true"  solution  in  Column  C  is  not 
available.   In  addition,  it  may  be  impractical  or  impossible  to  make  the 
adjustments  suggested  by  Column  D  in  table  3,  i.e.,  can  Hll  actually  increase 
teaching  outputs  to  nearly  three  times  its  recent  level?  This  suggests  that 
the  manager  might  use  the  DEA  results  to  identify  the  presence  of  inefficiency 
and  specific  areas  where  the  inefficiencies  may  lie.  The  DEA  results  also 
suggest  alternative  paths  to  improve  efficiency,  e.g.,  in  the  above  case,  Hll 
could  either  emulate  H4  or  H7  or  it  could  aim  for  the  composite  input-output 
level  suggested  in  table  3.  The  identification  of  the  preferred  and 
attainable  paths  to  improve  efficiency  would  naturally  be  based  on  managerial 
judgment.  Should  this  lead  to  proposed  paths  that  differ  from  the  one  derived 
from  DEA,  it  is  also  possible  to  reapply  DEA  for  a  sensitivity  analysis  to 
determine  if  other  paths  proposed  by  management  would  improve  the  efficiency 
compared  to  the  other  DMU's  in  the  data  set. 

4.  Field  Application  of  DEA  to  teaching  hospitals 

DEA  was  used  to  evaluate  efficiency  a  group  of  seven  teaching  hospitals  in 
Sherman  [14].  The  seven  hospitals  in  the  study  were  within  a  group  of 
teaching  hospitals  that  were  identified  as  being  "comparable"  based  on  cluster 
analysis  completed  by  the  state  rate  setting  agency  and  based  on  further 
review  and  adjustment  which  also  reflected  the  opinion  of  hospital 
administrators  about  what  constitutes  "comparable"  groupings.  The  state 
auditors  developed  a  set  of  ratios  to  identify  hospitals  whose  operating  costs 
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were  at  a  level  which  suggested  that  they  were  excessively  costly  and  possibly 

poorly  managed.  Hospitals  falling  into  this  "relatively  inefficient"  category 

were  then  subject  to  an  in-depth  audit  to  determine  if  these  costs  were 

justifiable  [10], 

DEA  was  applied  solely  to  the  medical  surgical  area  of  hospitals  using  the 

following  outputs  produced  with  the  following  inputs  during  one  year. 

Outputs  Inputs 

y^         Patient  days  of  care  -  age  >  65        x^  Full  time  equivalent  of 

nonphysician  medical 
surgical  staff 

y2    Patient  days  of  care  -  age  <  65         X2  medical  surgical  staff 

supply  dollars 

y3    Number  of  nursing  students  in  X3  Bed  days  available 

training  during  year 

y4    Number  of  interns  and  residents  in 
training  during  the  year 

These  outputs  and  inputs  were  selected  in  cooperation  with  a  panel  of 

hospital  experts  to  reasonably  represent  the  key  inputs  and  outputs  in  this 

part  of  the  hospital  as  described  in  [14].  The  data  was  collected  from  the 

1976  reports  submitted  to  the  state  regulatory  agency  by  each  hospital. 

The  DEA  results  identified  two  of  the  seven  hospitals  as  inefficient. 
There  exists  no  objective  rating  of  hospital  operations  covering  these 
hospitals.   Hence,  to  verify  these  results,  a  panel  of  hospital  experts 
familiar  with  these  hospitals  was  asked  to  evaluate  the  DEA  results.   In 
addition,  the  results  were  discussed  with  management  of  hospital  D,  one  of 
the  two  inefficient  hospitals.  These  procedures  (fully  described  in  [14]) 
confirmed  that  the  DEA  results  reflected  real  inefficiencies  and  in  hospital 
D,  for  example,  DEA  specifically  located  an  inefficient  level  of  bed  days 
and  personnel  used  to  produce  the  given  level  of  outputs. 

The  experts  agreed  that  the  two  hospitals  identified  were  likely  to  be 
relatively  inefficient  and  that  the  DEA  results  were  reasonable.  Management 
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of  hospital  D  reviewed  the  results  and  agreed  that  their  personnel  levels 
and  bed  days  available  were  somewhat  higher  than  comparable  hospitals. 
Hospital  management  reduced  its  bed  days  but  has  maintained  its  high 
personnel  level  for  internal  policy  reasons. 

The  value  of  DEA  in  this  context  is  best  illustrated  by  comparison  with 
the  auditor's  system  of  locating  potentially  inefficient  hospitals.  Table  4 
summarizes  the  DEA  results  and  lists  two  of  the  key  ratios  used  by  the 
auditor  to  locate  hospitals  that  were  potentially  inefficient  and, 
therefore,  candidates  for  further  review.  The  state  audit  agency  had  only 
identified  hospital  D  as  potentially  inefficient  because  it  was  the  only 
hospital  in  this  group  with  costs  over  one  standard  deviation  above  the  mean 

Table  4 
Comparison  of  teaching  Hospitals*  Medical  Surgical  area 


Hospital** 

DEA 
Efficiency 
Rating 

Efficiency 
Reference 
Set  (ERS) 

P( 

MS  Cost 
2r  Patient 
Day 

MS  Cost 
Per  Patient 

A 

1.0 

- 

$34 

!|;408 

B 

1.0 

- 

38 

418 

C 

1.0 

- 

39 

429 

D 

0.88 

A,  D,  0 

32 

407 

E 

1.0 

- 

27 

343 

F 

1.0 

- 

29 

348 

6 

0.93 

0 

36 

324 

Average  Cost 

^34. 

29 

$382.43 

Standard  Deviation 

4. 

n 

42.51 

cost  per  patient  day  and  cost  per  patient.  DEA  did  not  locate  D  as  ineffi- 
cient; however,  it  accurately  identified  hospital  D  and  G  as  inefficient,  an 
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insight  not  available  to  the  auditor  using  financial  ratios.  Although  these 
two  hospitals  were  not  the  most  costly,  there  still  existed  Inefficiencies 
located  by  DEA  which,  if  remedied,  could  make  them  less  costly.   Thus,  DEA 
captures  efficiency  dimensions  not  covered  in  the  financial  ratio/analytic 
review  technique  used  by  the  auditors.  These  inefficiencies  would  not 
generally  be  located  through  the  use  of  financial  ratios  because  such  ratios 
cannot  consider  multiple  outputs  simultaneously.   For  example,  a  hospital  that 
provides  a  high  volume  of  teaching  services  would  have  higher  operating  costs 
which  would  translate  into  augmented  patient  cost.  Teaching  outputs  and 
patient  care  are  jointly  produced  outputs  with  the  same  set  of  inputs  and  the 
costs  cannot  readily  be  separated  to  determine  the  cost  per  teaching  output 
versus  the  cost  per  patient.  Hence,  the  ratio  used  tends  to  ignore  certain 
outputs  which,  in  this  case,  is  the  teaching  output. 

5.  Use  of  DEA  to  evaluate  efficiency  of  for  profit  organizations 
A  study  of  savings  and  loan  bank  branches  suggests  that  DEA  may  be  useful 
for  managerial  audits  in  a  for  profit  setting  within  a  single  business. 
Sherman  and  Gold  [15]  applied  DEA  to  the  fourteen  branches  of  a  savings  bank 
to  attempt  to  locate  relatively  inefficient  branches.  While  the  savings  bank 
had  certain  profitability  criteria,  it  had  no  clear  concept  of  the  amount  of 
resources  that  were  required  to  produce  the  transactions  that  branches  were 
responsible  for.  Many  bank  transactions  require  resources  that  were  not 
directly  correlated  with  profit  measures.  For  example,  the  withdrawal  of 
funds  from  a  savings  account  reduces  the  funds  available  for  loan  or 
investment  and,  therefore,  reduces  profitability.  Nevertheless,  withdrawals 
are  in  a  sense  one  of  the  outputs  for  which  a  branch  is  responsible,  and  if 
this  and  other  transactions  could  be  completed  more  efficiently,  the  resource 
cost  might  be  reduced  and  profits  increased.   The  branch  profit  measure  used 
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by  the  bank  was  not  sensitive  enough  to  address  the  transaction  efficiency  so 
the  DEA  approach  was  applied  to  evaluate  its  potential  in  this  setting. 

Based  on  discussions  with  bank  management,  the  inputs  were  defined  to  be 
full-time  equivalents  of  branch  personnel,  supply  costs,  and  rent.  Rent  was 
used  as  rough  surrogate  for  square  feet  of  space  due  to  unavailability  of  the 
latter.-  (This  was  understood  to  be  a  surrogate  for  certain  branches  and 
visual  tests  of  the  results  were  performed  to  assure  that  this  did  not  bias 
the  results).  The  outputs  were  developed  by  segregating  the  numerous 
transactions  completed  at  a  branch  into  four  levels  of  complexity  such  that 
the  most  complex  transactions  were  these  which  were  likely  to  be  most  time 
consuming.   (See  [15]  for  details.) 

Of  the  fourteen  branches,  six  were  found  to  be  inefficient  using  DEA.  One 
of  these  six  was  closed  by  management  because  it  was  not  believed  to  be 
yielding  adequate  profits,  without  regard  to  the  DEA  results.  Four  of  the  six 
were  believed  to  be  relatively  weak  in  operations  and,  therefore,  the  DEA 
results  were  consistent  with  management's  "gut  feel"  but  much  more  explicit. 
The  other  inefficient  branch  was  believed  to  be  a  strong  branch  and  management 
is  investigating  this  to  determine  why  the  highly  profitable  branch  is  also 
technically  inefficient.  In  contrast  to  the  above  examples,  this  study  did 
not  attempt  to  validate  the  DEA  results,  but  rather  assumes  DEA  to  be  reliable. 

This  application  of  DEA  indicates  how  an  internal  audit  group  might  adopt 
DEA  to  evaluate  business  operation  where  there  are  several  offices  with 
similar  outputs  operating  in  different  locations,  with  different  output 
levels,  and  with  different  mixes  of  outputs  and  inputs.  Other  examples  where 
DEA  may  be  useful  are  commercial  bank  branches,  insurance  claims  offices, 
insurance  brokerage  offices,  customer  service  networks  supplying  field  service 
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networks  to  support  the  parent  company  products,  or  multiple  offices  of  a  CPA 
firm. 

One  may  question  whether  a  (for  profit)  business  should  use  DEA,  when  the 
measures  like  profitability  and  return  on  investment  are  available  and  are 
widely  acknowledged  as  key  indicators  of  performances  in  contrast  to  the 
nonprofit  case  where  such  measures  are  less  relevant.  There  appears  to  be  at 
least  two  justifications  for  use  of  DEA  in  the  for  profit  context.   First,  the 
financial  ratios  and  analytic  review  techniques  are  highly  dependent  on  use  of 
dollar  measures  which  can  be  biased  because  of  inflationary  factors,  regional 
price  differences,  or  methods  of  accounting  for  costs.  DEA  provides  a  means 
to  evaluate  performance  based  on  physical  units  of  inputs  and  outputs  and  can, 
therefore,  suggest  which  DMU's  are  efficiently  using  their  inputs  without 
respect  to  the  price  they  pay  for  these  inputs  and  without  regard  to  the  cost 
accounting  methods  applied.  As  was  illustrated  in  the  hospital  application, 
financial  ratios  may  provide  valid  insights  about  profitability  and  costs,  but 
DEA  can  provide  additional  insights  about  inefficiencies  which,  if  remedied, 
may  further  Increase  profitability  through  improved  efficiency. 

The  second  reason  DEA  seems  appropriate  in  the  for  profit  setting  is  that 
the  profit  and  ROI  measures  tend  to  reflect  current  operation  and  may  even  be 
penalized  by  expenditures  for  training,  maintenance  and  repair,  research  and 
development  which  are  incurred  to  promote  future  profitability.   These  other 
expenditures  may  result  in  outputs  which  are  not  reflected  in  current  sales 
and  are,  therefore,  not  considered  in  the  profit  measure.   These  same  outputs 
can,  however,  be  included  in  a  DEA  evaluation.  The  application  of  DEA  to 
evaluate  savings  bank  branches  represents  the  first  application  to  a  (for 
profit)  business  and  it  served  to  clarify  certain  problems  and  benefits  that 
can  be  derived  from  such  application.   The  true  potential  will  be  better 
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understood  in  application  to  other  business  organizations  such  as  are  listed 
above  and  which  applications  are  now  in  the  formative  stage. 

6.  Conclusion  -  PEA  as  a  Managerial  audit  tool 

Based  on  the  studies  described  in  this  paper  and  on  the  theoretical 
development  of  DEA  in  [4]  [5]  and  [6],  DEA  appears  to  be  a  technique  that  is 
well  suited  for  efficiency  evaluations  in  certain  managerial  audit  contexts. 
Our  findings  also  suggest  that  DEA  results  are  valid  but  with  important 
qualifications  which  need  to  be  considered  in  interpreting  the  results.  I 
first  summarize  where  DEA  appears  to  be  a  useful  managerial  audit  tool  and 
follow  with  a  summary  of  the  strengths  and  limitations  observed  in  the  DEA 
results.  Finally,  areas  of  further  research  to  improve  and  expand  the 
usefulness  of  DEA  for  managerial  audits  are  proposed. 

How  DEA  can  be  used  for  Managerial  Audits 

DEA  has  been  shown  to  be  capable  of  achieving  the  following  objective: 

Location  of  relatively  inefficient  DMU's  among  a  set 
of  DMU's  that  use  multiple  inputs  to  produce  multiple 
outputs  where  the  efficient  production  function  is  not 
known,  i.e.,  where  inputs  and  outputs  can  be  observed  and 
measured,  but  the  efficient  amount  of  inputs  required  for 
each  output  is  not  known  or  readily  determinable. 

This  type  of  information  may  be  useful  in  a  managerial  audit  context  for 
two  purposes:   1)  audit  resource  allocation  and  2)  analytical  review  of 
operations. 

Audit  resource  allocation.  DEA  can  identify  the  relatively  inefficient 
DMU's  which  are  most  likely  to  benefit  from  deeper  investigation  to  pinpoint 
the  source  of  these  inefficiencies  as  a  basis  for  developing  recommendations 
for  their  reduction.  In  this  context,  DEA  is  an  attention  direction  tool  to 
allocate  the  limited  managerial  audit  resources  to  areas  where  operating 
inef ficiences  are  known  to  exist. 
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Analytic  review  of  operations*  Where  auditors  are  evaluting  a  set  of 
DMU's  to  identify  ways  in  which  operations  can  be  improved,  DEA  can  be  used  to 
locate  the  relatively  more  and  less  efficient  DMU's  as  a  basis  for  studying 
and  understanding  the  managerial  techniques  used  in  more  efficient  units  and 
transferring  these  techniques  to  less  efficient  units. 

As  suggested  in  section  4,  DEA  can  serve  as  a  useful  complement  to 
traditional  financial  ratio  analysis,  and  that  it  may  prove  more  powerful  than 
ratio  analysis  where  the  simultaneous  consideration  of  multiple  outputs  and 
inputs  is  required  for  an  efficiency  assessment  to  achieve  either  of  the  above 
managerial  audit  purposes. 

DEA  is  particularly  applicable  to  public  sector  and  nonprofit  organization 
applications  because  they  not  only  have  multiple  outputs  and  inputs,  but  also 
because  their  outputs  can  often  be  measured  only  in  terms  of  physical  units 
and,  therefore,  are  not  readily  convertible  to  dollar  output  measures  as  are 
needed  for  many  financial  ratios.  DEA  is  also  of  potential  value  in  a  for 
profit  setting  where  either  of  the  following  conditions  exist: 

1.  Outputs  produced  by  a  firm  are  not  adequately  incorporated  into 
the  profit  measure  because  they  represent  investments  in  people  or 
products  that  are  expected  to  improve  the  future  earning  power  of 
the  firms,  e.g.,  employee  training,  product  development,  customer 
relations  development.   In  this  case,  DEA  may  serve  as  a 
performance  measure  that  reflects  performance  related  to  a  longer 
time  period  than  that  reflected  in  the  profit  and  ROI  measures. 

2.  The  desired  evaluation  of  firms  performance  with  financial  ratios 

does  not  adequately  consider  their  use  of  physical  inputs  to 

produce  their  outputs.  The  savings  bank  example  discussed  in 

section  5  is  an  example  of  how  DEA  can  focus  on  operating 

efficiency  as  distinct  from  overall  profitability  as  a  way  of 

locating  operating  improvements  that  may  ultimately  lead  to 

increased  profitability. 

-20- 


Strengths  and  Limitations  of  PEA 

DEA  has  been  shovra  to  be  theoretically  sound  and  consistent  with 
economic  theory  [4]  and  [5].  In  addition,  tests  of  DEA  suggest  that  it 
provides  reliable  information  in  an  application  to  a  relatively 
uncomplicated  hypothesized  set  of  production  relationships  and  for  a  set  of 
teaching  hospitals.  While  these  tests  of  DEA  and  the  underlying  theoretical 
soundness  do  not  necessarily  assure  its  reliability  in  all  field 
applications,  they  do  provide  insights  not  readily  available  from  a  study 
confined  to  the  technique  and  its  mathematical  underpinning.  These  findings 
may  be  summarized  as  follows: 

-  DMU's  identified  as  inefficient  with  DEA  are  strictly  inefficient 
compared  with  other  DMU's  in  the  data  set.  In  addition,  when  inefficient 
DMU's  are  located,  they  are  found  to  be  inefficient  with  respect  to  a 
narrower  set  of  relatively  efficient  DMU's  which  helps  to  focus  the 
investigation  into  the  source  and  nature  of  the  inefficiencies. 

-  DEA  will  not  necessarily  locate  all  the  relatively  inefficient  units 
and  at  this  point,  we  have  no  reliable  method  of  determining  how 
comprehensive  DEA  is  in  locating  inefficient  DMU's. 

-  DEA  identifies  alternative  paths  for  improving  the  efficiency  of  an 
inefficient  DMU  up  to  the  level  of  the  relatively  efficient  DMU's  in  the 
data  set.  DEA  does  not,  however,  identify  the  one  path  that  will  move  the 
inefficient  DMU  to  the  underlying  efficient  production  relationship.  Hence, 
managerial  judgment  is  required  to  assess  the  improvement  paths  which  are 
most  appropriate  for  a  particular  DMU. 
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-  DEA  has  been  tested  using  physical  output  and  Input  measures  and 
consequently  has  been  limited  in  its  use  to  assessment  of  technical 
output-input  efficiency.   Hence,  when  inefficiencies  are  identified,  DEA 
indicates  that  the  same  outputs  could  have  been  produced  with  fewer  inputs 
than  were  used.   DEA  does  not,  however,  address  issues  of  whether  a  firm  is 
purchasing  inputs  at  the  lowest  price  or  whether  the  input  mix  results  in 
the  lowest  cost  of  producing  a  good  or  service.  While  it  may  well  be 
possible  to  incorporate  relative  prices  into  the  DEA  model,  this  has  not  yet 
been  tested  and  requires  further  research.  (In  addition,  the  use  of 
nonphysical  input  and  output  measures  may  further  weaken  the  results.  For 
example,  use  of  an  index  of  quality  or  complexity  of  outputs  as  an  output 
measure  has  been  shown  to  lead  to  misleading  results  in  [15]). 

In  general,  the  use  of  DEA  for  managerial  audit  purposes  should  be 
limited  to  the  identification  of  inefficient  DMU's  and  the  general  magnitude 
of  these  inefficiencies  based  on  use  of  physical  output  and  input  measures. 
Before  expanding  the  use  of  DEA  beyond  this  for  managerial  audit  purposes, 
these  other  uses  need  to  be  carefully  researched  for  their  theoretical 
soundness  as  well  as  for  their  performance  in  field  tests. 

Future  research  into  use  of  DEA  as  a  managerial  audit  tool 

While  much  work  is  in  process  to  expand  the  underlying  DEA  model  as 
reflected  for  example  in  [4]  and  [5],  there  are  a  number  of  application- 
oriented  areas  of  research  which  may  help  to  improve  DEA's  capabilities  in 
managerial  audit  applications  which  are  briefly  described  below. 

Output  and  Input  identification  and  measurement 
The  procedures  for  identifying  relevant  outputs  and  inputs  for  a  DEA 
efficiency  assessment  are  critical  to  the  relevance  of  the  results.  When 
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relevant  outputs  and  Inputs  are  either  ignored  or  unmeasurable,  the  DEA 
results  can  be  biased  and  possibly  misleading.  Further  research  to  understand 
how  to  effectively  identify  and  measure  these  outputs  and  inputs  and  the 
effects  of  errors  in  omission  is  needed.  This  will  be  particularly  important 
in  extending  DEA  to  the  for  profit  setting  where  the  multiple  outputs  may  have 
not  beep  readily  identified  and  measured  in  physical  terms,  due  to  the  heavy 
reliance  on  financial  measures  of  performance. 

Introduction  of  relative  prices  -  Testing  of  DEA  with  the  introduction  of 
relative  price  constraints  in  the  linear  program  model  may  extend  the  results 
to  evaluate  price  efficiency  and  input  mix  (allocative)  efficiency  as  well  as 
the  already  encompassed  technical  efficiency.  This  would  expand  the 
capabilities  of  DEA  in  the  nonprofit  applications  and  may  make  DEA 
substantially  more  attractive  and  powerful  in  for  profit  application  where 
information  about  relative  competitive  market  prices  is  readily  available. 

Use  of  DEA  in  a  time  series  as  well  as  cross-section  application.  The  DEA 
applications  to  date  concentrate  on  a  cross-section  of  DMU's  operating  in  one 
period.  It  may  be  possible  to  use  multiyear  or  multiquarter  data  to  detect 
trends  in  relative  efficiency  over  time  which  may  sharpen  the  managerial  audit 
applications.  For  example,  it  may  be  possible  to  identify  DMU's  that  were 
relatively  efficient  and  which  have  recently  become  relatively  inefficient  and 
vice  versa  which  would  allow  the  audit  to  focus  on  dynamic  as  well  as  static 
sources  of  inefficiencies. 

In  conclusion,  existing  tests  of  DEA  support  its  use  in  managerial  audits 
of  organization  efficiency  when  interpreted  in  the  ways  suggested  in  this 
paper.  While  greater  insights  may  also  be  available  from  DEA,  these  insights 
must  be  carefully  subject  to  theoretical  and  field  tests  of  the  validity. 


-23- 


REFERENCES 

[I]  Burton,  J.C.  and  P.  Fairfield,  "Auditing  Evolution  In  a  Changing 
Environment",  Auditing,  Winter,  1982,  Vol,  1,  No.  2,  pp  1-22. 

[2]   Bessent,  A.  and  W.  Bessent,  "Determining  the  Comparative  Effilcency  of 
Schools  Through  Data  Envelopment  Analysis,"  Research  Report  CCS  361 
(Austin,  Texas:  Center  for  Cybernetic  Studies,  University  of  Texas  at 
Austin,  December  1979). 

[3]   Bessent,  A.  and  W.  Bessent,  J.  Kennington  B.  Regan,  "An  Application  of 
Mathematical  Programming  to  Assess  Productivity  in  the  Houston 
Independent  School  District,"  Technical  Report  OR  80011  (Dallas, 
Texas:  Southern  Methodist  University,  June  1980). 

[4]   Charnes,  A.,  W.W.  Cooper,  A.  Lewln,  R.  Morey  and  J.  Rousseau, 

"Efficiency  Analysis  with  Non-discretionary  Resources,"  Research  Report 
379  (Austin,  Texas:  Center  for  Cybernetic  Research,  University  of  Texas 
at  Austin,  September  1980). 

[5]   Charnes,  A.,  W.W.  Cooper,  and  E.  Rhodes,  "Measuring  Efficiency  of 

Decision  Making  Units,"  European  Journal  of  Operations  Research,  Vol.  3, 
No.  4,  July  1979,  p.  339. 

[6]    Charnes,  A.,  W.W.  Cooper,  and  E.  Rhodes,  "Evaluating  Program  and 

Managerial  Efficiency:  An  Application  of  Data  Envelopment  Analysis  to 
Program  Follow  Through,"  Management  Science,  Vol.  27,  No.  6,  June  1981, 
pp.  668-697. 

[7]   Churchill,  H.C.,  W.W.  Cooper,  V.  Govindarajan,  J.D.  Pond,  and  J.G.  San 
Miguel,  "Developments  in  Comprehensive  Auditing  and  Suggestions  for 
Research"  Symposium  on  Auditing  Research  II  (Department  of  Accounting, 
University  of  Illinois  at  Urbana  Champaign,  1977). 

[8]   Lewln,  A.Y.  and  R.C.  Morey,  "Measuring  the  Relative  Efficiency  and 

Output  Potential  of  Public  Sector  Organizations:   An  Application  of  Data 
Envelopment  Analysis  Journal  of  Policy  Analysis  and  Information 
Systems,"  Vol.  5,  No.  4,  December  1981. 

[9]   Macchiaverna,  P.,  Internal  Auditing,  (The  Conference  Board,  1978). 

[10]  Massachusetts  Rate  Setting  Commission,  Hospital  Burearu,  "Grouping  of 

Hospitals  for  Purposes  of  Determining  Reasonable  Cost  Levels,"  Document 
No.  114.1  CMR,  Rate  Setting  Commission,  1980. 

[II]  Mosher,  F.C.,  The  GAP;  The  Quest  for  Accountability  in  American 
Government  (Boulder,  Colorado:  Westview  Press,  1979). 

[12]   San  Miguel,  J.  and  R.  Banker,  "Portland  General  Electric  Company", 
Harvard  Business  School  Case  #       ICCH,  Cambridge,  MA. 

[13]   Sherman,  H.D.,  "Identifying  Inefficiencies  in  Multiple  Output-Multiple 
Input  Organizations,"  SSM  Working  Paper  //1316-82. 


-24- 


[14]   Sherman,  H.D.,  Measurement  of  Hospital  Technical  Efficiency:   A 
Camparative  evaluation  of  Data  Envelopment  Analysis  and  Locating 
Inefficiency  in  Health  Care  Organizations,"  unpublished  Doctoral 
Dissertation  Harvard  Graduate  School  of  Business  Administration,  1981. 

[15]  Sherman,  H.  D,,  and  F  .  Gold,  "Evaluating  Operating  Efficiency  of 

Service  Businesses  with  Data  Envelopment  Analysis  -  Empirical  Study  of 
Bank  Branch  Operations,"  Sloan  School  of  Management  Working  Paper 
//1444-83,  June  1983. 

[16]   Banker,  R.,  A.  Charnes,  W.W.  Cooper  and  A.  Schinnar,  "Extremal  Principle 
for  Frontier  Estimation  and  Efficiency  Evaluation",  Management  Science. 

[17]   "Statement  in  Auditing  Standards  //23:  Analytical  Review  Procedures", 
AICPA,  1978. 


-25- 


435 


040 


3    TDflD    DDM    523    T05 


Date  Due 


AUG.  3''1  f906 
^»1  9 


iq# 


F£ 


''  ,  .-.  J:m 


Lib-26-67 


f 


B-^SEWENT 


^ 


>^^ 


a 


